Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/gov: Queries to /gov/proposals/{proposalID}/votes support pagination #5405

Merged
merged 5 commits into from
Dec 18, 2019

Conversation

dshulyak
Copy link
Contributor

@dshulyak dshulyak commented Dec 13, 2019

Changes:

  • votes for active proposals are paginated by applying page limits to results stored in keeper
  • for inactive proposals we paginate what we receive from tendermint.TxSearch

tendermint.TxSearch paginates transactions, not messages. I noticed that current gov application allows to submit only one msg per transaction. But transaction structure suggests that there could be many messages in a single transaction. Hence I load transactions in batches (30), unpacking them and once we collected enough votes - query is terminated. Or it will be terminated if we run out off transactions.

This approach gives clients what they want, but also adds unnecessary load. So if relying on application semantics is error-proof, let me know and i will simplify this part.

Other changes:

  • added separate type for proposal votes queries
  • extended cli for query gov votes with '--page' and '--limit' flags

Closes: #5344

Description


For contributor use:

  • Targeted PR against correct branch (see CONTRIBUTING.md)
  • Linked to Github issue with discussion and accepted design OR link to spec that describes this work.
  • Code follows the module structure standards.
  • Wrote unit and integration tests
  • Updated relevant documentation (docs/) or specification (x/<module>/spec/)
  • Added relevant godoc comments.
  • Added a relevant changelog entry to the Unreleased section in CHANGELOG.md
  • Re-reviewed Files changed in the Github PR explorer

For admin use:

  • Added appropriate labels to PR (ex. WIP, R4R, docs, etc)
  • Reviewers assigned
  • Squashed all commits, uses message "Merge pull request #XYZ: [title]" (coding standards)

@dshulyak
Copy link
Contributor Author

dshulyak commented Dec 13, 2019

There was a note that TxSearch doesn't support pagination. I assume it is outdated, right? Based on https://github.com/tendermint/tendermint/blob/master/rpc/core/tx.go#L79-L85

@codecov
Copy link

codecov bot commented Dec 13, 2019

Codecov Report

Merging #5405 into master will decrease coverage by 0.22%.
The diff coverage is 91.89%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #5405      +/-   ##
==========================================
- Coverage   54.59%   54.37%   -0.23%     
==========================================
  Files         315      313       -2     
  Lines       18921    18601     -320     
==========================================
- Hits        10330    10114     -216     
+ Misses       7810     7712      -98     
+ Partials      781      775       -6
Impacted Files Coverage Δ
x/gov/keeper/querier.go 45.18% <85.71%> (+1.77%) ⬆️
x/gov/client/utils/query.go 24.36% <93.33%> (ø)
x/params/subspace/subspace.go 0% <0%> (-77.36%) ⬇️
x/params/subspace/table.go 74.35% <0%> (-25.65%) ⬇️
x/bank/handler.go 69.04% <0%> (-16.67%) ⬇️
types/dec_coin.go 63.21% <0%> (-15.39%) ⬇️
x/params/proposal_handler.go 62.96% <0%> (-10.73%) ⬇️
crypto/keys/keybase_base.go 64.7% <0%> (-3.72%) ⬇️
client/keys/utils.go 45.45% <0%> (-3.16%) ⬇️
crypto/keys/keyring.go 43.65% <0%> (-1.9%) ⬇️
... and 58 more

x/gov/client/utils/query_test.go Show resolved Hide resolved
} {
t.Run(tc.description, func(t *testing.T) {
params := types.NewQueryProposalVotesParams(0, tc.page, tc.limit)
votes, err := getPaginatedVotes(context.CLIContext{}, params, makeQuerier(tc.txs))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using the variable on range scope tc in function literal (from scopelint)

t.Run(tc.description, func(t *testing.T) {
params := types.NewQueryProposalVotesParams(0, tc.page, tc.limit)
votes, err := getPaginatedVotes(context.CLIContext{}, params, makeQuerier(tc.txs))
if tc.err != nil {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using the variable on range scope tc in function literal (from scopelint)

@dshulyak
Copy link
Contributor Author

dshulyak commented Dec 16, 2019

Will be great if someone will be able to review this change. I noticed golangcibot warnings, and will fix them with other suggestions. Also coverage seems to report irrelevant info, reports coverage decrease in not related modules and files.

Copy link
Contributor

@alexanderbez alexanderbez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Amazing! Thanks for the contribution @dshulyak -- I left some feedback, but overall this is the correct idea!

x/gov/client/cli/query.go Show resolved Hide resolved
@@ -72,41 +73,57 @@ func QueryDepositsByTxQuery(cliCtx context.CLIContext, params types.QueryProposa
return cliCtx.Codec.MarshalJSON(deposits)
}

type txQuerier func(cliCtx context.CLIContext, events []string, page, limit int) (*sdk.SearchTxsResult, error)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure why the function was broken up this way? It's a bit harder to grok for me.

Supporting pagination for this one-off case can be achieved with minimal changes to QueryVotesByTxQuery. Let me know what you think.

func QueryVotesByTxQuery(cliCtx context.CLIContext, params types.QueryProposalParams) ([]byte, error) {
	events := []string{
		fmt.Sprintf("%s.%s='%s'", sdk.EventTypeMessage, sdk.AttributeKeyAction, types.TypeMsgVote),
		fmt.Sprintf("%s.%s='%s'", types.EventTypeProposalVote, types.AttributeKeyProposalID, []byte(fmt.Sprintf("%d", params.ProposalID))),
	}

	var (
		votes        []types.Vote
		searchResult *sdk.SearchTxsResult
		err          error
	)

	page := 1

	searchResult, err = utils.QueryTxsByEvents(cliCtx, events, page, 100)
	if err != nil {
		return nil, err
	}

	// Search for votes until the tx results are exhausted or we've reached the
	// pagination limit.
	for searchResult.Count != 0 || len(votes) < params.Limit {
		for _, info := range searchResult.Txs {
			for _, msg := range info.Tx.GetMsgs() {
				if msg.Type() == types.TypeMsgVote {
					voteMsg := msg.(types.MsgVote)

					votes = append(votes, types.Vote{
						Voter:      voteMsg.Voter,
						ProposalID: params.ProposalID,
						Option:     voteMsg.Option,
					})
				}
			}
		}

		page++

		searchResult, err = utils.QueryTxsByEvents(cliCtx, events, page, 100)
		if err != nil {
			return nil, err
		}
	}

	start, end := client.Paginate(len(votes), params.Page, params.Limit, 100)
	if start < 0 || end < 0 {
		votes = []types.Vote{}
	} else {
		votes = votes[start:end]
	}

	if cliCtx.Indent {
		return cliCtx.Codec.MarshalJSONIndent(votes, "", "  ")
	}

	return cliCtx.Codec.MarshalJSON(votes)
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i splitted whole function so that i can actually test pagination logic (see query_test.go).
otherwise, body of the function is similar to yours, with one distinction - if the previous tx query returned less then full chunk we already know that tx results are exhaused and query terminated earlier

Copy link
Contributor

@alexanderbez alexanderbez Dec 17, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's keep the logic in QueryVotesByTxQuery because unit testing private functions is not always an option -- the end result is the same, you're still unit testing the same business logic.

You can update the above snippet to handle the case I missed 👍

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in order to unit test QueryVotesByTxQuery i need to mock calls to SearchTx, i do it with passing custom querier function. am i missing something that can be used for this purpose?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, you're right! We can add the function as a parameter to QueryVotesByTxQuery?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

imo bad api. but not worth arguing, if you feel that this is ok i will pass as param

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @dshulyak!

x/gov/client/utils/query_test.go Show resolved Hide resolved
@@ -53,6 +54,15 @@ func (keeper Keeper) GetVotes(ctx sdk.Context, proposalID uint64) (votes types.V
return
}

func (keeper Keeper) GetVotesPaginated(ctx sdk.Context, proposalID uint64, page, limit int) (votes types.Votes) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would remove this method and move the pagination logic within the querier handler like other modules do.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

@alexanderbez alexanderbez Dec 17, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I follow? This method is not needed. Pagination can simply be done in queryVotes.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i mean that when i was adding this method i intentionally made it consistent with that one (e.g. kept pagination in keeper)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. Yeah, but this isn't necessary in this case.

x/gov/keeper/querier.go Show resolved Hide resolved
Copy link
Contributor Author

@dshulyak dshulyak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@alexanderbez thank you for review. i responded to your comments, please take a look.
in gov module there are also deposits, do you think that it makes sense to add pagination for them as well?

x/gov/client/cli/query.go Show resolved Hide resolved
x/gov/client/utils/query_test.go Show resolved Hide resolved
@@ -53,6 +54,15 @@ func (keeper Keeper) GetVotes(ctx sdk.Context, proposalID uint64) (votes types.V
return
}

func (keeper Keeper) GetVotesPaginated(ctx sdk.Context, proposalID uint64, page, limit int) (votes types.Votes) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@@ -72,41 +73,57 @@ func QueryDepositsByTxQuery(cliCtx context.CLIContext, params types.QueryProposa
return cliCtx.Codec.MarshalJSON(deposits)
}

type txQuerier func(cliCtx context.CLIContext, events []string, page, limit int) (*sdk.SearchTxsResult, error)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i splitted whole function so that i can actually test pagination logic (see query_test.go).
otherwise, body of the function is similar to yours, with one distinction - if the previous tx query returned less then full chunk we already know that tx results are exhaused and query terminated earlier

Changes:
- votes for active proposals are paginated by applying page limits to results stored in keeper
- for inactive proposals we paginate what we receive from tendermint.TxSearch
tendermint.TxSearch paginates transactions, not messages. I noticed that current gov application allows to
submit only one msg per tx. But tx structure suggests that there could be many messages in a single transaction.
Hence we load transactions in batches (30), unpacking them and once we collected enough votes - query is terminted.
Or it will be terminted if we run out off transactions.

Other changes:
- add separate type for proposal votes queries
- extended cli to query gov vote with '--page' and '--limit' flags

Signed-off-by: dmitry <[email protected]>
@dshulyak dshulyak changed the title x/gov: Queries to /gob/proposals/{proposalID}/votes support pagination x/gov: Queries to /gov/proposals/{proposalID}/votes support pagination Dec 18, 2019
@dshulyak
Copy link
Contributor Author

  • removed pagination from keeper
  • and QueryVotesByTxQuery now accepts TxQuerier

@alexanderbez let me know if you have other concerns

@dshulyak
Copy link
Contributor Author

btw GolangCI chocked on downloading tendermint, would be great if someone can restart

Running error: context loading failed: failed to load program with go/packages: -: go: downloading github.com/tendermint/tendermint v0.32.8

Copy link
Contributor

@alexanderbez alexanderbez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good @dshulyak. Only thing remaining is to not return an error in one case.


start, end := client.Paginate(len(votes), params.Page, params.Limit, 100)
if start < 0 || end < 0 {
return nil, fmt.Errorf("invalid page (%d) or range is out of bounds (limit %d)", params.Page, params.Limit)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
return nil, fmt.Errorf("invalid page (%d) or range is out of bounds (limit %d)", params.Page, params.Limit)
votes = []type.Vote{}

x/gov/client/utils/query_test.go Outdated Show resolved Hide resolved
Copy link
Contributor

@alexanderbez alexanderbez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support Pagination for Governance Votes
3 participants